Method for generating a training data set for training an artificial intelligence module for a control device of a vehicle

ABSTRACT

A method for generating a training data set for training an artificial intelligence (AI) module. An image sequence is provided in which surroundings of a robot are recorded. A trajectory in the recorded surroundings is determined. At least one future image sequence is generated which extends to a time segment in the future, and, based on the at least one determined trajectory, encompasses a prediction of images for the event that the determined trajectory was followed during the time segment in the future. At least one sub-section of the determined trajectory in the generated image sequence is assessed as positive or as negative when a movement predicted by following the trajectory corresponds to a valid movement situation, or as an invalid movement situation, respectively. The generated future image sequence with the assessment assigned thereto of the trajectory are combined for generating a training data set for the AI module.

FIELD

The present invention relates to the field of artificial intelligence,in particular, to a method for generating a training data set fortraining an artificial intelligence module, or AI module. The AI modulemay, for example, be implemented in a control device for a device movingin an at least semi-automated manner, such as an at leastsemi-autonomous robot or a vehicle driving in an at least semi-automatedmanner.

BACKGROUND INFORMATION

In automation technology, robotics, in autonomous driving etc., forexample, artificial intelligence modules, hereafter also referred to asAI modules, are frequently used for the automated controlling of, e.g.,at least semi-autonomous robots. These are based on trained data and areto ensure, e.g., during the at least semi-autonomous actual operation ofthe robot, a control that takes into consideration, in particular,surroundings of the robot, e.g., a roadworthy control in the case of amotor vehicle driving in an at least semi-automated manner, in that theyinitiate suitable responses for occurring driving events. With respectto the vehicle engineering, e.g., the vehicle is to be controlled insuch a way that collisions with obstacles and/or other road users areprevented or the motor vehicle closely follows the track of thecontinuously changing course of the roadway.

For this purpose, such an AI module may include at least one artificialneural network, for example, which is also referred to as ANN hereafter.This ANN is trained using training data sets to gradually teach the AImodule how to move, e.g., drive, autonomously in a roadworthy manner.

However, a simulator is not yet available which is able to sufficientlyrealistically map the surroundings and, in particular, possible robotsurroundings, e.g., vehicle surroundings, to teach an, in particular,simulated robot how to move safely, e.g., to teach a motor vehicle howto drive in a roadworthy manner. For example, the computing effort wouldbe comparatively high for such a realistic simulator since, based on thevehicle engineering, at least roads, static and dynamic objects, andalso the movement behavior of the dynamic objects would have to besimulated. In practice, the surroundings are therefore reduced orsimplified to a model based on which a simulation for training the AImodule is created. For this purpose, for example, an ANN may be trained,which reduces the surroundings to such a simplified model. It has beenfound that the training success for the AI module to be trained usingthis reduced simulation is in need of improvement.

SUMMARY

Specific embodiments of the present invention provide an option for animproved training of an AI module as well as a use of the AI modulebased thereon. Advantageous refinements of the present invention arederived from the description herein, as well as the figures.

The described example method according to the present invention issuitable for generating a training data set for training an artificialintelligence module, or AI module. As an alternative or in addition, thedescribed method may also be suitable for training an AI module. The AImodule may, for example, be a software program for a computer-assistedcontrol device of a robot, e.g., of a motor vehicle driving in an atleast semi-automated manner. The AI module may be configured to generatean output for electronically activating the robot by a control device,e.g., by a control unit of the motor vehicle driving in an at leastsemi-automated manner, and to supply it to the motor vehicle, which isable to ascertain an evasive maneuver and/or a braking maneuver, forexample, based on the output. The control device may furthermore promptthe robot, e.g., the motor vehicle, to carry out this evasive maneuverand/or braking maneuver by the activation of actuators or the like. Forthis purpose, the AI module may include program code and also, inparticular, multilayer and/or convolutional artificial neural networks(ANN).

The example method according to the present invention may be implementedin a computer-assisted manner in, e.g., a data processing unit, whichmay also include at least one memory unit and one processing unit, andincludes the following steps:

-   -   Initially, an image sequence is provided in which, in general,        surroundings, in particular, robot surroundings, are recorded in        images. In other words, the image sequence encompasses images of        surroundings or an environment in which the robot may be        present, move, etc. The image sequence may be recorded in        advance with the aid of a vehicle, e.g., which includes an image        recording unit, such as a camera, LIDAR sensors etc. In the        actual driving operation, this unit may be driven through        different surroundings and may create one or multiple image        sequence(s), which may be provided here as an image sequence for        the described method.    -   Then, at least one trajectory is determined, which is situatable        in the robot surroundings. In this connection, a trajectory may        be understood to mean a kind of space curve, i.e., a possible        track or path of the robot or of another, in particular, dynamic        object in or through the environment or vehicle surroundings        present in the image sequence. The trajectory may be considered        to be situatable when it may be considered as implementable,        e.g., with respect to physical boundaries, by an assigned        object, i.e., an object which is to follow the trajectory. In        other words, at least one arbitrary trajectory is generated for        all arbitrary objects which result from the provided image        sequence of the robot surroundings recorded therein. For        example, on the one hand, all possible movements (except for the        limitation of the accuracy, a finite number of trajectories,        i.e., different trajectory configurations) of dynamic objects        may be taken into consideration, but also the movement of the        robot itself relative to its surroundings.    -   At least one future, in particular, artificial image sequence is        generated, which extends to a time segment in the future with        respect to a sequence ending point in time and, based on the at        least one determined trajectory, encompasses a prediction of        images for the event that the determined trajectory was followed        during the time segment in the future. In other words, one or        multiple future image sequence(s) is/are artificially generated        based on selected possible trajectories. The image sequence may        also be considered as a simulation, in particular, a short-term        simulation, in which the entire scene which results based on the        image sequence and/or the prediction is simulated. This        simulation may encompass all dynamic and non-dynamic components        of the image sequence and/or of the robot surroundings recorded        therein.    -   Then, an assessment of at least one sub-section of the        determined trajectory included in the generated image sequence        takes place. A result of the assessment or the assessment is        positive when a movement predicted by following the trajectory        corresponds to a valid movement situation, or is negative when        the movement predicted by following the trajectory corresponds        to an invalid movement situation. In other words, it is        estimated based on the generated image sequence how features        identifiable therein, such as roadway markings, slopes, or        objects, such as other road users, etc., may change, move, or        possibly represent an obstacle, in particular, with respect to        the trajectory. For example, a dynamic object could move in the        prediction from a first point A to a second point B situated        along the trajectory, and then represent a potential obstacle.        In the prediction, however, the trajectory could also be tangent        to or intersect a roadway marking, such as a shoulder of a road.        Accordingly, the predicted movement situation may be a collision        with another static or dynamic object, a veering off a road, or        similar situations. Based on an application in vehicle        engineering, e.g., a movement situation may be understood to        mean a driving event, a driving situation, etc.    -   For generating a training data set for the AI module to be        trained, the generated future image sequence is then combined        with the assessment of the trajectory assigned thereto to        generate a training data set for the AI module. This means that        the training data set is based on established data in the form        of the image sequence or the generated future image sequence        based thereon, in combination with the assessed prediction for a        time segment which goes beyond the image sequence, i.e., a        future time segment.

With this configuration, the described example method according to thepresent invention enables an improved training of an AI module since ituses a “pseudo” or “offline” driving simulator, “pseudo” or “offline”being intended to indicate here that the training data are based on arecorded image sequence of the actual surroundings, and not on a puresimulation which is simplified compared to the actual surroundings. Dueto the contact with reality compared to a simplified simulation, a hightraining quality may be achieved. In particular, one trajectory ormultiple trajectories may be determined for each object identified inthe robot surroundings. The realistic scenes based thereon, in the formof images, are ideally, depending on quality, possibly indistinguishablefrom the provided, actually recorded image sequence. Since thetrajectories of the individual objects in the robot surroundings areknown as a result of the determination, a considerably improvedprediction is also possible. In this way, the quality of theartificially generated image sequences may also be further improved inthe future for the time segment. At the same time, the computing effortfor training the AI module may be kept low since a comprehensivesimulation of the vehicle surroundings is no longer required, but only aprediction for a comparatively short time segment, which accordinglyrequires less computing effort.

In one particularly advantageous refinement of the present invention, itis provided that the training data set is fed into the AI module.Thereafter, the generated training data set may serve as an inputvariable for an ANN, in particular, for an input layer thereof, and/orfor a learning algorithm of the AI module, which utilizes, e.g., anapproach of machine learning, such as reinforcement learning, supervisedlearning, etc. Due to the contact with reality of this training dataset, the learning success of the AI module may occur more quickly. As aresult of a processing and/or an interpretation of the training dataset, the neural network of the AI module may ascertain, provide and/oroutput an output.

To utilize the image sequence preferably efficiently, a single imagesequence may be provided, from which a plurality of training data setsare generated, determining trajectories which each differ from oneanother. This means that the above-described method steps which followthe provision of the image sequence are carried out repeatedly, againusing this one image sequence. Thus, different trajectories aregradually determined, a respective prediction is made beyond the imagesequence, the trajectories are assessed based on the respectiveprediction, and these findings, as described above, are combined to formtraining data or are generated from this combination. It is possible,e.g., that different geometric variables of the originally determinedtrajectory are gradually varied, i.e., for example, an angle, acurvature, etc. For this purpose, a variation of the trajectory of adynamic object identified in the provided image sequence and/or of therobot, e.g., vehicle, from whose perspective the image sequence isgenerated, may be carried out, for example based on different steeringangles at a constant speed, for example in 5° or, in particular, in 1°increments. At the same time, or as a further variable, it is alsopossible to include possible speed changes in the selection of thetrajectories, for example the described steering angle changes, atsimultaneous delays by a realistic delay value (for example based on theinstantaneous driving situation, as a function of the robot, e.g.,vehicle, its speed, and outside circumstances, such as a wetness of aroadway). In this way, multiple possible trajectories are created, whichmay all form the basis for the subsequent prediction of the artificiallygenerated image sequences. It is also possible to change all furthertrajectories of moving objects in this way, and to incorporate them inthe prediction. In this way, a finite number of trajectoryconfigurations are created, which may all cause different predictions,so that different image sequences may be predicted or generated for eachtrajectory configuration.

Accordingly, in one further advantageous refinement of the presentinvention, at least one first training data set may be generated fromthe provided image sequence, based on a first determined trajectory, anda second training data set may be generated, based on a seconddetermined trajectory. It is also possible, of course, to generate stillfurther training data sets beyond this, the further trajectories, whichtogether with their assessment and the image sequence are combined toform a further training set, differing from preceding trajectories in atleast one feature and/or one property of the trajectory.

According to one refinement of the present invention, the generatedimage sequence for the respective determined trajectory may encompass anumber of depth images, real images and/or images of a semanticsegmentation along the same trajectory. In other words, the output foreach trajectory configuration may be a number of image sequences ofdepth images, real images and/or images of a semantic segmentation alongthe same trajectory. In this way, it is possible to generateparticularly realistic training data for different scenarios and/or fortraining different sensors, etc.

In one refinement of the present invention, the trajectory for a dynamicobject included in the provided image sequence may be determined, and,based thereon, the future image sequence may be generated. In otherwords, a dynamic object may be identified in the provided imagesequence, for which initially one or multiple differenttrajectory/trajectories is/are determined, and for which the futureimage sequences including the corresponding prediction along theparticular trajectory are generated. When the trajectories of theindividual objects are known, this results in a considerable improvementof the prediction, i.e., the artificially generated image sequences inthe future.

According to one refinement of the present invention, the trajectory forthe robot may be determined and, based thereon, the future imagesequence may be generated.

To avoid unnecessary computing effort during the determination of thetrajectory, a preselection of the possible trajectories may be made. Thepreselection of the trajectory may, for example, preferably take placebased on the traffic situation, taking a predetermined probabilitydistribution into consideration. For example, taking the instantaneousvehicle speed into consideration, which may be computationallyascertained, e.g., from the image sequence, trajectories may bediscarded as unrealistic when they, based on a learned or definedprobability distribution, are not situatable in the environment orvehicle surroundings recorded in the image sequence in images. Atrajectory which, e.g., requires a physically not implementable lateralguidance force, which may, for example, also be determined based on avehicle dynamics model or another computational consideration, wouldthus be unlikely. This trajectory would not be taken into considerationin the preselection based on the probability distribution. In this way,it is possible to achieve that the method is only carried out for suchtrajectories which result in a qualitatively useful training data set.

As an alternative or in addition, the determination of the trajectorymay also take place by a random selection thereof based on apredetermined probability distribution. In this way, a plurality oftrajectories may be randomly taken into consideration, the selectionbeing limited to those trajectories which are at least largelyrealistically implementable based on the probability distribution in thesurroundings or environment of the robot predefined by the imagesequence. In this way, it is possible to save computing effort since themethod is only carried out for such trajectories which result in aqualitatively useful training data set.

The trajectories to be determined may be preselected in that thedetermination only takes place for that trajectory or those trajectorieswhich is/are implementable based on the driving situation, taking avehicle dynamics model into consideration. It is possible, for example,that a trajectory having a curved progression in the vehicle plane has aradius which is not implementable for vehicles in general, or somevehicle types, in terms of vehicle dynamics, e.g., because the curvenegotiation resulting along the trajectory can physically not beimplemented. Such a trajectory may then be discarded even prior to thedetermination, without having to run through the method up to theassessment of the trajectory which, consequently, is only assessable asnegative. The vehicle dynamics model may, e.g., be the so-called circleof forces, it also being possible to use more detailed models.

Surprisingly, it has been found that a comparatively short time segmentfor the prediction is already sufficient to achieve a good quality ofthe training data set and, at the same time, keep the computing effortlow. Accordingly, in one advantageous refinement of the presentinvention, the time segment, beginning with or after the sequence endingpoint in time, for the prediction may be established with a durationbetween 0.5 s and 1.5 s, preferably with approximately 1 s. Thisduration has proven to be a good compromise between the quality of thetraining and the required computing effort. Moreover, it is possible touse common image processing methods having a sufficiently goodprediction accuracy for such a time period.

One advantageous refinement of the present invention provides that theprediction includes at least one or multiple of the following methods:monocular depth estimation, stereo depth estimation, LIDAR dataprocessing and/or estimation from optical flow. From the optical flow ofmultiple individual images of the image sequence, e.g., a prediction ora forecast for further individual images, which are no longer includedin the image sequence beyond the sequence ending point in time, may takeplace. Depending on the method, not all individual images of the imagesequences necessarily have to be processed, but only a subset thereof.These methods for depth prediction are, e.g., also available as atoolbox and may thus be procured easily and used for this purpose.

To be able to classify objects, such as obstacles or other road users,and/or features of the vehicle surroundings recorded in the imagesequence, the prediction may include the generation of a semanticsegmentation of at least several individual images of the imagesequence. In this way, both the driving events may be predicted moreaccurately, and their assessment may be improved. Of course the methodsof depth prediction and of semantic segmentation may be combined, sothat the prediction accuracy may be improved yet again. The estimationfrom the optical flow may thus, for example, only be carried out forcertain classes of the semantic segmentation, such as for dynamicobjects.

A further advantageous refinement of the present invention providesthat, during the assessment, an object recognition and/or a featurerecognition obtained from the semantic segmentation is used to weightthe positive or negative assessment obtained from the assessment. If theassessment, without a weighting, on an exemplary scale were, e.g., zeroor −1 for a negative assessment and, e.g., +1 for a positive assessmentof the predicted driving event, it is possible to distinguish betweendifferent driving events based on the object and/or feature recognition.For example, a collision with an object classified as a pedestrian couldbe assessed more negatively, e.g., with −1, than a driving over a curb,which could, e.g., have the value −0.7. The assessment of the lessconsequential collision with the curb would, in this example, thus beless negative in absolute terms than the collision with the pedestrian.This object and/or feature recognition may thus further improve thetraining quality of the AI module.

The example method according to the present invention is not onlysuitable for trajectories for an ego-vehicle from whose perspective theimage sequence, or the vehicle surroundings recorded therein, isrecorded when entering it. One advantageous refinement of the presentinvention, for example, provides that the determination of at least onetrajectory, the prediction of a driving event for the time segment, andthe assessment of the trajectory based on the prediction are carried outfor a dynamic object identified in the image sequence, which isconsequently different from the ego-vehicle. In this way, a movingpedestrian may be identified as a dynamic object, e.g., with the aid ofsemantic segmentation and/or prediction. Analogously to what wasdescribed above, at least one trajectory is then determined for thispedestrian, the prediction and assessment are carried out for thistrajectory, and a training data set is also generated therefrom.

One particularly advantageous refinement of the present inventionprovides that the method described here uses the approach of theso-called reinforcement learning for training the AI module.Reinforcement learning is a conventional methodology of machine learningin which the above-described assessment is also regarded as a positiveor negative reward.

As an alternative to the approach of reinforcement learning, the examplemethod described herein, however, may also take place based on theapproach of the so-called supervised learning, which, in turn, is aconventional methodology of machine learning. According to thisrefinement of the present invention, a positively assessed trajectorymay be supplied to a supervised learning algorithm as a valid drivingsituation. For example, a positively assessed trajectory may beprocessed together with the prediction as a kind of sequence/label pairfor a valid driving situation.

The present invention also relates to a data processing unit which mayalso include, e.g., at least one memory unit as well as a computingunit, and is configured to carry out the above-described method.

The present invention furthermore also relates to a device forcontrolling, in particular, a control device, at least onesemi-autonomous robot, the device being configured to carry out a methoddescribed herein to select an assessed trajectory therefrom, and toactivate the robot according to the selected trajectory.

Further measures improving the present invention are described hereafterin greater detail together with the description of the preferredexemplary embodiments of the present invention based on the figures.

BRIEF DESCRIPTION OF THE DRAWINGS

Exemplary embodiments of the present invention are described in detailhereafter with reference to the accompanying figures.

FIG. 1 shows a schematic overview of an application of an example methodaccording to the present invention for training an AI module.

FIG. 2 shows a flow chart for illustrating steps of an example methodaccording to the present invention for training an AI module.

The figures are only schematic representations and are not true toscale. In the figures, identical, identically acting or similar elementsare consistently denoted by identical reference numerals.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

For better illustration, FIG. 1 shows a schematically representedapplication 100 of an example method according to the present inventionfor training an artificial intelligence module 1, which is referred tohereafter in short as AI module 1. It includes one or multiplemultilayer ANN(s), for example, which is able to generate an output inthe form of signals for a device for controlling an at leastsemi-autonomous robot. This output may prompt the device for controllingthe robot to activate actuators and similar units to automatically carryout computer-controlled movements. The robot here, only by way ofexample, is a vehicle driving in an at least semi-automated manner. Asan alternative thereto, the at least semi-autonomous robot may also beanother mobile robot (not shown), for example one which moves by flying,swimming, diving or walking. The mobile robot may, for example, also bean at least semi-autonomous lawn mower or an at least semi-autonomouscleaning robot. One or multiple actuator(s), e.g., a drive and/or asteering system of the mobile robot, may also be electronicallyactivated in these cases in such a way that the robot moves at leastsemi-autonomously.

Hereafter the robot is only described as a vehicle by way of example. Asis described hereafter, AI module 1 of the vehicle is trained using aplurality of training data sets 2, which are supplied to an input layerof the ANN, for example.

In an optional step S0 (see also the flow chart in FIG. 2), the methodaccording to the present invention for training AI module 1 initiallyprovides that at least one image sequence 5 of vehicle surroundings 6 iscreated during the actual driving operation, e.g., with the aid of avehicle 3 under real driving conditions, i.e., in the actual drivingoperation of vehicle 3 used for the training, with the aid of an imagerecording unit 4. For this purpose, image recording unit 4 includes,e.g., one or multiple camera(s), LIDAR sensor(s) etc. Image sequence 5is then present as a video file, e.g., and may be processed accordinglyin a computer-assisted manner.

In a step S1 (see also the flow chart in FIG. 2), image sequence 5,together with vehicle surroundings 6 included therein, is provided forfurther processing, image sequence 5 having a sequence starting point intime, i.e., the beginning of the recording of vehicle surroundings 6,and a sequence ending point in time to, i.e., the end of the recordingof vehicle surroundings 6. Vehicle surroundings 6 here are a public roadby way of example, e.g., a rural road, and include, by way of example,multiple static and dynamic objects 7, 8, 9 as well as multiple features10, 11, 12, which are a road user in the form of another vehicle(=dynamic object 7), two trees at the left and right roadsides (=staticobjects 8, 9), and the roadway center and roadside markings (=features10, 11, 12). For better illustration, vehicle 3 (outside vehiclesurroundings 6) is also shown as an ego-vehicle here, with which imagesequence 5 was established prior to its provision as a video file, andfrom whose perspective vehicle surroundings 6, shown in a simplifiedmanner here, are represented.

The provided image sequence 5 is then further processed, e.g., with theaid of a data processing unit 13, which is shown by way of example ingeneral terms in FIG. 1 as a processor or a workstation including atleast one memory unit as well as a computing unit, using correspondingcomputing instructions. The further steps of the method described here,which are described hereafter, are also carried out with the aid of dataprocessing unit 13.

In a step S2, for example, at least one trajectory 14 a, 14 b, 14 c, 14d situatable in the provided vehicle surroundings is determined by,e.g., a traffic situation-based, vehicle dynamics-dependent and/ordriving situation-based selection, taking a learned or definedprobability distribution into consideration. Accordingly, a preselectionof the trajectories is preferably made here, trajectories which areunrealistic based on the driving situation, e.g., being discarded priorto the determination in step S2. In the example shown in FIG. 1, thefour trajectories 14 a, 14 b, 14 c, 14 d are determined by way ofexample since these [are] situatable in predefined vehicle surroundings6 and generally also negotiable based on the driving situation, i.e.,taking, for example, physical boundary conditions into consideration.

In this exemplary embodiment, trajectory 14 a, which begins at theego-vehicle here, intersects the presumable roadway of dynamic object 7,which is additionally also determined as trajectory 14 d here.Trajectory 14 e also begins at object 7 and leads past roadside marking11 into, e.g., a slope of the shown road. Beginning at the ego-vehicle,trajectory 14 b continues straight ahead in the same obstacle-free lane.Having the same origin, trajectory 14 c leads toward object 10, i.e., astatic object.

For each of these trajectories 14 a through 14 e, at least one futureimage sequence is generated in a step S3 (see also the flow chart inFIG. 2), which extends to a time segment t0+n in the future with respectto sequence ending point in time t0 and, based on the at least onedetermined trajectory 14 a through 14 e, encompasses a prediction ofimages for the event that the determined trajectory 14 a through 14 ewas followed, either by the ego-vehicle or, in the case of trajectories14 d and 14 e, by dynamic object 7. This means that, based on thedynamic objects, including the ego-vehicle, those trajectories 14 athrough 14 e are calculated which are possible for these objects, i.e.,possible in terms of vehicle dynamics. These trajectories 14 a through14 e represent a kind of parameterization to form the image sequences ofa prediction into the future, either using AI or further conventionalprediction methods. The output for each of trajectories 14 a through 14e are artificially generated image sequences of depth images, realimages and/or images of a semantic segmentation along the sametrajectory 14 a through 14 e in each case. In this way, the entire sceneof the particular image sequence is simulated, a preferably large numberof, or all, dynamic and non-dynamic components of the robot surroundingsrecorded in image sequence 5 being included. The simulation of the imagesequence takes place in different ways, namely as a semanticsegmentation including multiple images, as an actual image includingmultiple images and/or as a depth image including multiple images.

However, only a comparatively short time segment t0+n is considered inthe process, which begins at or after sequence ending point in time t0and extends from there into the future by, e.g., 0.5 s to 1.5 s.Depending on the accepted computing effort, however, it is also possibleto predict longer time segments using the prediction methods explainedhereafter so that the considered time segment t0+n may also be extended.In this exemplary embodiment, time segment t0+n, however, is establishedat 1 s, which has proven to be advantageous for numerous practical caseswith respect to the computing effort and achievable benefit.

For the prediction of the image sequence, the provided image sequence 5is further processed using suitable image processing methods and/orprediction methods, which are available as software packages or thelike, in a computer-assisted manner in data processing unit 13, usingthe determined trajectories 14 a through 14 e as a kind of parameter.

To make a preferably exact prediction for future movement situations,such as driving events, a classification of objects 7, 8, 9 and offeatures 10, 11, 12 may take place into, e.g., these two classes or,even more precisely, into the classes vehicle, tree, roadway center, androadside marking. This classification may also encompass suitablemethods for the semantic segmentation, which are generally conventionaland may be pixel- or voxel-based, e.g., similar regions, i.e., forexample, all adjoining pixels, in terms of content, of the particularobject 7, 8, 9 being combined into regions which are coherent in termsof content.

As an alternative or in addition to the semantic segmentation, aprediction of the images suitable for this purpose and/or a depthprediction is/are carried out, for example, based on image sequence 5.The depth prediction preferably includes a monocular or stereo depthestimation, an estimation from the optical flow and/or a LIDARestimation, e.g., through the use of a Kalman filter, based on severalor all individual images of image sequence 5 or similar methods. Thedepth prediction uses, e.g., an ANN, in particular, an autoregressiveconvolutional neural network, which autoregressively makes a predictionfor time segment t+n from the given individual images of image sequence5 beyond sequence ending point in time to. From a practical point ofview, image sequence 5 at sequence ending point in time t0 may serve asan input variable for a prediction at point in time t0+1, the predictionobtained therefrom at point in time t0+1 (which is already in theconsidered time segment t0+n and therefore is no longer included inimage sequence 5) may, in turn, serve as an input variable for a furtherprediction at point in time t0+2, etc., to make a prediction until pointin time t0+n, i.e., across the entire time segment to be predicted.Moreover, however, other prediction methods are also possible, whosedescription at this point is dispensed with for the sake of clarity. Apossible change of the vehicle surroundings along and/or adjoining therespective trajectory 14 a through 14 e is thus estimated for timesegment t0+n.

In this exemplary embodiment, the result of the forecast or of theprediction may be that the driving event related to trajectory 14 a is acollision with the moving, i.e., dynamic, object 7. This is easilycomprehensible based on FIG. 1 since trajectories 14 a and 14 d shown asdotted lines intersect there. Trajectory 14 d determined for object 7,considered on its own, continues straight ahead and, in principle,signifies an undisturbed continued travel for object 7. With respect toego-vehicle 3, however, trajectory 14 d, as explained above, intersectstrajectory 14 a of ego-vehicle 3 and thus presumably leads to thecollision therewith. However, it shall be noted that the explicitdetermination of trajectory 14 d is not absolutely necessary for thisprediction, but the movement of object 7 may also be estimated directlyfrom the prediction. In contrast, the predicted driving event based ontrajectory 14 b would, in principle, be an undisturbed continued travel.In contrast, the predicted driving event based on trajectory 14 c wouldbe a collision with static object 9.

In a step S4 (see also the flow chart in FIG. 2), the respectivegenerated image sequence and/or at least a sub-section of the determinedtrajectory 14 a through 14 e included therein is/are assessed. Therespective sub-section or the entire trajectory 14 a through 14 e isassessed as positive when the predicted movement situation, e.g., thepredicted driving event, (cf. steps S3 and S4, see also FIG. 2)corresponds to a valid movement situation, e.g., valid drivingsituation. In contrast, the respective sub-section or the entiretrajectory 14 a through 14 e is assessed as negative when the predictedmovement situation, e.g., the predicted driving event, corresponds to aninvalid movement situation, e.g., invalid driving situation.

In this exemplary embodiment, trajectory 14 a, or the following ornegotiation thereof, is assessed negatively as an invalid drivingsituation due to the collision with object 7 predicted therefor.Trajectory 14 c is also assessed negatively as an invalid drivingsituation since here again a collision with object 9 is predicted.Trajectory 14 e based on object 7 is also to be assessed as negativesince feature 11, as a boundary marking of the roadway, is driven overhere, and object 7 would veer off the roadway in this case. However,trajectory 14 b is assessed positively for the ego-vehicle, andtrajectory 14 d is assessed positively for object 7, in which clearstraight-ahead driving is to be expected, and this corresponds to avalid driving situation.

It is notable that the assessment takes place in a positively andnegatively weighted manner, i.e., may also be relativized. In thisexemplary embodiment, trajectory 14 a, due to the severity of thecollision with another road user (=object 7), is assessed morenegatively than trajectory 14 c, which, while it also results in acollision, does not, e.g., affect another road user, or possibly mayalso offer a longer stopping distance etc. Accordingly, the respectiveassessment of the driving event may thus be weighted according to thedegree of the particular validity or invalidity of the drivingsituation.

In a step S5 (see also the flow chart in FIG. 2), the artificiallygenerated image sequence including the prediction from step S3 for theparticular trajectory 14 a through 14 e is combined with the respectiveassessment from step S4 using corresponding computing instructions indata processing system 13. Training data set 2 is then generated with orfrom this combination, which is consequently made up of, in principle,established data, namely image sequence 5, and the image sequence basedthereon including the prediction for the particular trajectory 14 athrough 14 e for time segment t0+n as well as the respective assessmentthereof. Training data set 2 generated from this combination thuscorresponds to a kind of pseudo-driving simulator, which is based onreal vehicle surroundings 6 of image sequence 5 and encompassesartificially generated image sequences, in which preferably manydifferent movements of dynamic objects, including ego-vehicle 3 movingrelative to the surroundings, are taken into consideration. The outputfor each trajectory 14 a through 14 e are image sequences of depthimages, real images and/or images of a semantic segmentation along thesame trajectories.

In an optional step S6 (see also the flow chart in FIG. 2), thistraining data set 2 from step S5 is supplied to AI module 1 as, e.g.,input variable(s) of its ANN, i.e., for example, its input layer, or itsother learning algorithm and is fed there. It is provided in the processthat training data set 2 is used for machine learning using an approachfor reinforcement learning, to train the ANN of AI module 1 using thisapproach.

FIG. 2 shows the sequence of the method according to the presentinvention including optional step S0, steps S1 through S5, and optionalstep S6 in the form of a flow chart. It is notable that this trainingmethod takes place for AI module 1, as explained above, based onrecorded image sequences in combination with a prediction, obtained fromthe image sequence, for a predetermined time segment t0+n going beyondthe image sequence.

Proceeding from the shown embodiment, the method according to thepresent invention may be modified in many respects. It is possible, forexample, that, in optional step S6, the assessment is not used for theabove-described reinforcement learning, but is used together with theartificially generated image sequence or prediction as a sequence/labelpair for a valid driving situation for training a supervised learningalgorithm. It is furthermore possible that the above-described method iscarried out in real time by a device for controlling a robot, e.g., acontrol unit in a vehicle etc., and based on the assessment of thedifferent trajectories which are assessed based on the above-describedmethod, a trajectory is selected, and the robot is electronicallyactivated for moving according to the selected trajectory.

1-15. (canceled)
 16. A method for generating a training data set fortraining an artificial intelligence (AI) module, comprising thefollowing steps: providing an image sequence in which surroundings of arobot are recorded; determining at least one trajectory which issituatable in the recorded surroundings of the robot; generating atleast one future image sequence which extends to a time segment in thefuture with respect to a sequence ending point in time, and, based onthe at least one determined trajectory, encompasses a prediction ofimages for an event that the determined trajectory was followed duringthe time segment in the future; assessing at least one sub-section ofthe determined trajectory included in the generated image sequence aspositive when a movement predicted by following the determinedtrajectory corresponds to a valid movement situation, or as negativewhen the movement predicted by following the determined trajectorycorresponds to an invalid movement situation; and combining thegenerated future image sequence with the assessment assigned to thedetermined trajectory for generating a training data set for the AImodule.
 17. The method as recited in claim 16, further comprising thefollowing step: feeding the training data set into the AI module. 18.The method as recited in claim 16, wherein only a single image sequenceis provided, and multiple trajectories which each differ from oneanother are generated from which a multitude of future image sequencesis generated.
 19. The method as recited in claim 16, wherein thegenerated image sequence for the determined trajectory encompasses anumber of: (i) depth images, and/or (ii) real images and/or (iii) imagesof a semantic segmentation, along the same trajectory.
 20. The method asrecited in claim 16, wherein the trajectory is determined for a dynamicobject included in the provided image sequence and, based the determinedtrajectory, the future image sequence is generated.
 21. The method asrecited in claim 16, wherein the trajectory is determined for the robotand, based on the determined trajectory, the future image sequence isgenerated.
 22. The method as recited in claim 16, wherein, prior to thedetermination, a preselection of the trajectory situatable in thesurroundings is made, taking a predetermined probability distributioninto consideration.
 23. The method as recited in claim 16, wherein thedetermination is only made for the one trajectory or multipletrajectories which are implementable based on a driving situation,taking an assigned vehicle dynamics model of the robot configured as avehicle into consideration.
 24. The method as recited in claim 16,wherein the time segment is established with a duration between 0.5 sand 1.5 s.
 25. The method as recited in claim 16, wherein the timesegment is established with a duration of 1 s.
 26. The method as recitedin claim 16, wherein the prediction includes at least one or multiple ofthe following methods: monocular depth estimation, stereo depthestimation, LIDAR data processing, and estimation from optical flow. 27.The method as recited in claim 16, wherein the prediction includesgeneration of a semantic segmentation of at least several individualimages of the image sequence.
 28. The method as recited in claim 27,wherein, during the assessment, an object recognition and/or a featurerecognition obtained from the semantic segmentation is used to weightthe positive or negative assessment.
 29. The method as recited in claim16, wherein the valid movement situation encompasses acollision-avoiding and/or road-following continued movement along thedetermined trajectory, and the invalid driving situation encompasses aveering off a roadway, a departure from a lane and/or a collision withanother object.
 30. A data processing unit for training an artificialintelligence module, which is configured to: provide an image sequencein which surroundings of a robot are recorded; determine at least onetrajectory which is situatable in the recorded surroundings of therobot; generate at least one future image sequence which extends to atime segment in the future with respect to a sequence ending point intime, and, based on the at least one determined trajectory, encompassesa prediction of images for an event that the determined trajectory wasfollowed during the time segment in the future; assess at least onesub-section of the determined trajectory included in the generated imagesequence as positive when a movement predicted by following thedetermined trajectory corresponds to a valid movement situation, or asnegative when the movement predicted by following the determinedtrajectory corresponds to an invalid movement situation; and combine thegenerated future image sequence with the assessment assigned to thedetermined trajectory for generating a training data set for the AImodule.
 31. A device for controlling an at least semi-autonomous robot,the device being configured to: provide an image sequence in whichsurroundings of a robot are recorded; determine at least one trajectorywhich is situatable in the recorded surroundings of the robot; generateat least one future image sequence which extends to a time segment inthe future with respect to a sequence ending point in time, and, basedon the at least one determined trajectory, encompasses a prediction ofimages for an event that the determined trajectory was followed duringthe time segment in the future; assess at least one sub-section of thedetermined trajectory included in the generated image sequence aspositive when a movement predicted by following the determinedtrajectory corresponds to a valid movement situation, or as negativewhen the movement predicted by following the determined trajectorycorresponds to an invalid movement situation; combine the generatedfuture image sequence with the assessment assigned to the determinedtrajectory for generating a training data set for the AI module; selectan assessed trajectory using the AI module; and activate the robotaccording to the selected trajectory.